Guilt is the result of years of doctrinal conditioning and mind control. The feeling is common among people who break free from high control groups.
Many people experience the feelings of guilt even outside of a scenario like you are having with your family.
Part of the healing process is getting past these feelings of guilt. The way to do that is by focusing on rationalisation and logic, as well as realising that the feeling of guilt is a normal emotion to experience when you are exiting a religion like the JWs.
The intensity of what you are feeling right now is what keeps many people locked within the organisation.
The good news is it does get easier, as you learn to distinguish what emotions you should be feeling from the ones that have been imposed on you through indoctrination.
It is not selfish to take back your mind from these people, even if you experience the feeling that it is. Don't forget that they selfishly stole your mind in the first place with lies and deceit.